CALL GLOP()

Andy Malakov software blog

Saturday, November 23, 2024

Commissioning new solar inverter into SunPower PVS5

I have SunPower solar panels on my roof and have been using their monitoring app to periodically check on my system. This summer, however, SunPower filed for bankruptcy, which prompted me to find an independent way to monitor historical production data. My solar setup also has SolarLog meter that operates independently and also used to have ability (3G Cellular) to reporting my solar production, but since 2022 (when cellular network providers phased out 3G), this feature has stopped working. With the 3G sunset, I lost access to historical production data via the SolarLog website. As a result, I decided to explore options for monitoring my production locally, while the SunPower interface is still operational (and I can validate my readings against theirs). I found an excellent guide by Dolf Starreveld on integrating the SunPower PVS6 module with Home Assistant. The guide details how to use a Raspberry Pi Zero W as a bridge between the PVS monitoring LAN and the home LAN. It also applies to the PVS5 SunPower solar supervisor unit, which features two network ports and integrates somewhat more easily than the newer PVS6, which rely on wireless connectivity.
What follows are my notes on how I successfully registered an additional inverter with the PVS5. I accessed the PVS5 management LAN using a Raspberry Pi Zero, but you should be able to follow these steps by connecting a laptop directly to the PVS5 LAN1 management interface as well. The management app should be accessible at http://172.27.153.1 or, if your laptop obtains an IP address via the PVS5 DHCP server, at http://www.sunpowerconsole.com.
DISCLAIMER: These steps are provided "as is" for informational and educational purposes only. I am not responsibile or liabe for any errors, damages, or issues that may arise from following these instructions. Use at your own risk. Always exercise caution when working with electricity.
If you need to update your PVS5 solar configuration (such as adding an inverter), you’ll need to run the "commission" setup process using the SunPower PVS5 Management App.

Step 1: Verify access to SunPower PVS Management App

If you are connected directly to the PVS management port, the SunPower PVS Management app should be accessible at https://172.27.153.1. If you are using a Raspberry Pi as a proxy, connect to the Raspberry Pi’s IP address within your local network. The interface should look like this:

Step 2: Tick the SunPower PVS Management app to work with your own server.

This app is built with Angular and interacts with the SunPower server at https://pvsmgmt.us.sunpower.com. However, as of November 2024, the server endpoints used by the app are no longer functional and return 503 errors. If you inspect the app’s network activity using Google Chrome Developer Tools, you’ll see that it makes the following calls to the main server:

  • https://pvsmgmt.us.sunpower.com/PvsMgmtAPI/api/v1/device/logFwCalls
  • https://pvsmgmt.us.sunpower.com/PvsMgmtAPI/api/v2/rma/candidates
  • https://pvsmgmt.us.sunpower.com/PvsMgmtAPI/api/v1/device/savePvsConfig
  • https://pvsmgmt.us.sunpower.com/PvsMgmtAPI/api/v1/commissioning/config
  • https://pvsmgmt.us.sunpower.com/PvsMgmtAPI/api/v1/module/types

Based on what I observed most of these calls are simply for logging purposes, except for the last one. That call is used to retrieve solar panel and inverter models from the SunPower database, based on the discovered inverter type and the user-provided panel wattage. For example, in my case, I have 335W panels and an AC_Module_Type_D microinverter. The server is expected to return a list of compatible panels with full model numbers, such as SPR-X21-335-BLK-C-AC, like this:

[
  {
    "Value": "SPR-X21-335-BLK-C-AC", # my panels
    "Type": "inverter-micro",
    "Key": 12345
  }
]

I am still unsure about the exact JSON payload the /module/types endpoint is supposed to return. It parses the AC/DC suffix of the panel model (provided in the Value result field) and verifies that the returned Type result field is "inverter-micro". The app also checks the Key result field and validates that it contains a number.

I made the following simple Python Flask program to simulate pvsmgmt.us.sunpower.com shown below.

from flask import Flask, jsonify, request
from flask_cors import CORS

app = Flask(__name__)
CORS(app) 

###########################################
## Stub for https://pvsmgmt.us.sunpower.com
###########################################

# Specific handler for /PvsMgmtAPI/api/v1/module/types
@app.route('/PvsMgmtAPI/api/v1/module/types', methods=['GET', 'POST'])
def module_types():
    filter_param = request.args.get('filter') # unused
    model_type = request.args.get('model_type')
    
    if model_type == "AC_Module_Type_D":     # TODO: Replace with your inverter type
        return jsonify([{
            "Value": "SPR-X21-335-BLK-C-AC", # TODO: Replace with full number model of your battery
            "Type": "inverter-micro",
            "Key": 12345
        }]), 200
    else:
        return jsonify({}), 200

#@app.route('/PvsMgmtAPI/api/v1/device/logFwCalls', methods=['GET', 'POST'])
#@app.route('/PvsMgmtAPI/api/v2/rma/candidates', methods=['GET', 'POST'])
#@app.route('/PvsMgmtAPI/api/v1/device/savePvsConfig', methods=['GET', 'POST'])
#@app.route('/PvsMgmtAPI/api/v1/commissioning/config', methods=['GET', 'POST'])
# Generic handler for all other /PvsMgmtAPI endpoints
@app.route('/PvsMgmtAPI/', methods=['GET', 'POST'])
def handle_requests(subpath):
    return jsonify({}), 200  # Empty / 200=success

if __name__ == '__main__':
    app.run(debug=True)
I run this flask program on default port 5000, and also run STUNNEL to wrap this in SSL using self-signed certificates.
[sunpower]
accept = 443
connect = 5000
cert = sunpower-cert.pem
key = sunpower-key.pem
verify = 0
sslVersion = TLSv1.2
options = NO_SSLv2
options = NO_SSLv3

Step 3: STUNNEL

Last bit was local DNS hack in my Windows system32\drivers\etc\hosts file:
# Sunpower hack
127.0.0.1 pvsmgmt.us.sunpower.com

Now when a page in my browser wanted to access https://pvsmgmt.us.sunpower.com it would go to local STUNNEL process on port 443, and from there to my Flask app running on port 5000.

CHECK: Refresh the PVS Management App and check Developer/Network panel in Chrome: you should see no errors when app tries to access https://pvsmgmt.us.sunpower.com.

Step 4: Run commissioning steps in PVS Management App

It may be helpful to familiarize yourself with the app using the SunPower Manual. Go to page 44 to learn how to use the app. Some of the appendices also provide useful information about additional features that were added to the app later. You may also find this manual generally helpful—for instance, it explains how to determine whether production tracking is set up on the LINE or LOAD side of your system.

Interestingly, the app contains an easter egg: if you encounter a black screen warning about the app’s sunset, you can bypass it by entering the Konami Code (Up, Up, Down, Down, Left, Right, Left, Right, B, A). In my case, the screen appeared, but I was able to dismiss it using the "Close" button. It seems that a thoughtful member of the SunPower development team enabled this lock screen dismissal.

After commissioning is successful, the app attempts to forward you to pvsmgmt.us.sunpower.com. According to the manual, at this step, you can edit general information about your site (such as the owner name and address). In my case, this step had already been completed during the initial system setup and was entirely unnecessary. The new inverter was registered successfully regardless.

That was it. Good luck if you want to follow these steps. Let me know if you need more information.

Wednesday, November 30, 2016

Wednesday, March 4, 2015

Connecting two CentOS computers using cheap Infiniband

This is a continuation of the previous post. This time I wanted to test direct Infiniband connection on Linux.

Setup is the same:

  • Two retired developer's desktops (built in 2008): AMD Opteron 2216 @2.4MHz, 8G DDR2.
  • A pair of Mellanox Infinihost III adapter MHGA28-XTC
  • CentOS Linux 6.3 (Minimal Install in my case)
  • MLNX_OFED_LINUX-1.5.3-4.0.42-rhel6.3-x86_64.iso OFED driver (Still available on mellanox.com)

Linux setup is pretty straightforward but in my opinion more involved than on Windows. Main problem was old age of these cars. In order to avoid rebuilding OFED drivers for these cards I used old version of CentOS (6.3). I've tried 2.x version but got MFE_OLD_DEVICE_TYPE error. Besides I wanted to test SDP in Java 7 and this protocol seems to be no longer available in OFED 2.x +.

Bottom line: for these old Infinihost III-family cards use older OFED driver (1.5.3). If you don't want to rebuild the driver, use Linux distro/version specified by the driver (there are quite a few).

I found that the following two resources most useful for this project: A and B. There is no reason to repeat these steps here. Connection verification and testing using OFED utilities is similar to Windows version.

Configuring simple Java Socket application to use SDP worked like a charm. See Oracle's tutorial.

Connecting two Windows 7 computers with low-cost Infiniband

Previous generation Infiniband cards are selling for a fraction of original price on eBay. Developers are buying these setups to test/learn this technology. I've followed this path and posting my notes here. The setup wasn't easy and I had to collect information from various sources.

Hardware

  • A pair of Mellanox Infinihost III adapter MHGA28-XTC ($36)
  • .
  • A pair of Molex 4X Infiniband copper cables ($12.5)

Total price tag was $97 (including shipping).

These are old-generation Dual-Port InfiniBand adapter cards that fit into PCI Express x8 slots. Each card has two 20Gb/s ports. I used two retired supermicro desktops (circa 2008) that fit these cards by age. Each computer is running Windows 7 (x64). [Next post will explore the same hardware setup on Linux].

Infinihost allows direct connection between two computers (in point-to-point setup there is no need for Infiniband switch).

BIOS Update

Multiple sources recommend upgrading card's firmware before trying them with Windows.

There are several revisions of MHGA28-XTC cards, mine was A3 (check the sticker attached to the back of each card). Firmware can be downloaded from Mellanox here.

To upgrade firmware and basic status testing Mellanox provides MFT utilities set. In my case the latest MFT version 3.8 refused to work with these cards claiming they are no longer supported. Luckily MFT version 2.7.2 is still available and works with these old Infinihost-family cards:

C:\Program Files\Mellanox\WinMFT>mst status
MST devices:
------------
  mt25218_pciconf0
  mt25218_pci_cr0

C:\Program Files\Mellanox\WinMFT>mlxburn -dev mt25218_pci_cr0 -image fw-25218-5_3_000-MHGA28-XTC_A3.bin

    Current FW version on flash:  5.2.916
    New FW version:               5.3.0

Read and verify Invariant Sector            - OK
Read and verify PPS/SPS on flash            - OK
Burning second FW image without signatures  - OK
Restoring second signature                  - OK
-I- Image burn completed successfully.

Windows Driver

Initially these cards showed up as "Infiniband controller" in Windows Device Manager:

Driver for these cards are available from Mellanox and OpenFabrics.org (OFED). I believe both of these sources actually provide the same driver maintained by OFED (sponsored by Mellanox).

Here I had the same story - the latest OFED driver version (3.2) simply didn't work with these cards. Setup ended with "Possible NetworkDirect startup failure" warning. The installed driver would identify the card properly but yellow triangle said that device was disabled due to errors. Windows event log showed that some driver components failed to initialize.

After some trial and error I found that OFED driver version 2.3 was what I needed. I can be downloaded from OpenFabrics archive.

As you can see below, in addition to Infiniband card Device Managers showed that I got two OpenFabrics IPoIB Adapters (since each card has two ports):

Configuration

I repeating above steps on both computers and connected cards with cables.

OFED software comes with set of utilities, one of which (IBSTAT) can be used to check connectivity status:

C:\Windows\system32>ibstat
CA 'ibv_device0'
        CA type:
        Number of ports: 2
        Firmware version: 0x500030000
        Hardware version: 0x20
        Node GUID: 0x0002c9020023c250
        System image GUID: 0x0002c9020023c253
        Port 1:
                State: Initializing
                Physical state: LinkUp
                Rate: 20
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x90580000
                Port GUID: 0x0002c9020023c251
                Link layer: IB
        Port 2:
                State: Initializing
                Physical state: LinkUp
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x90580000
                Port GUID: 0x0002c9020023c252
                Link layer: IB

Subnet Manager

When these cards are connected directly we need to launch Infiniband Subnetwork Manager (opensm). OFED installs it as Windows Service (disabled by default). In my case I launched opensm from command line. You need to run this service on both computers.

C:\Windows\system32>opensm
-------------------------------------------------
OpenSM 3.3.6 UMAD
Command Line Arguments:
 Log File: %TEMP%\osm.log
-------------------------------------------------
OpenSM 3.3.6 UMAD

Entering DISCOVERING state

Using default GUID 0x2c9020023c251
Entering MASTER state

SUBNET UP

Entering STANDBY state

Each service can be configured to serve both ports (enter GUID of each port GUIDs into opensm configuration file).

After this step Windows should show your network status as connected:

If you plan to keep this setup running, OpenSM can be launched automatically as Windows Service (disabled by default).

Connectivity test

OFED has special ping utility that can be used for quick test.

Computer 1 (Here we print GUIDs of each port and launch ping server):

C:\Windows\system32>ibstat -p
0x0002c90200231745
0x0002c90200231746

C:\Windows\system32>ibping -S

Computer 2 (here we use GUID of the first computer's port):

C:\Windows\system32>ibping -G 0x0002c90200231745
Pong from ?hostname?.?domainname? (Lid 2): time 0.230 ms
Pong from ?hostname?.?domainname? (Lid 2): time 0.160 ms
Pong from ?hostname?.?domainname? (Lid 2): time 0.231 ms
Pong from ?hostname?.?domainname? (Lid 2): time 0.159 ms
Pong from ?hostname?.?domainname? (Lid 2): time 0.163 ms
Pong from ?hostname?.?domainname? (Lid 2): time 0.174 ms
(Nevermind weird host name).

Latency test

Computer 1 (Launching test server):
C:\Windows\system32>ib_send_lat -a -c RC 
Computer 2 (test client):
C:\Windows\system32>ib_send_lat -a -c RC oldfaithful
------------------------------------------------------------------
                    Send Latency Test
Inline data is used up to 400 bytes message
Connection type : RC
test
  local address:  LID 0x100, QPN 0x6040200, PSN 0x265a0000, RKey 0x2c0010 VAddr 0x00000001170040
  remote address: LID 0x200, QPN 0x6040600, PSN 0xb64e0000, RKey 0x2c0030 VAddr 0x00000000fc0040
Mtu : 2048
------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
      2        1000           4.10        2295.99             4.27
      4        1000           3.75        1305.61             4.27
      8        1000           3.75         256.86             3.93
     16        1000           3.75         266.24             3.93
     32        1000           4.44        1021.62             4.61
     64        1000           4.44         329.22             4.61
    128        1000           4.61         303.79             4.78
    256        1000           4.95         529.41             5.12
    512        1000           5.46         300.89             5.63
   1024        1000           6.83         309.25             7.00
   2048        1000           9.22         327.17             9.39
   4096        1000          11.61         280.92            11.78
   8192        1000          16.90         306.86            17.07
  16384        1000          27.65         329.56            27.82
...
This hardware is 9 years old, so numbers are sub-optimal. Still much better than TCP even without any special tuning.

What's next?

We got low-latency 20Gb/s connection between two Windows 7 machines using a pair of cheap Infiniband adapters.

In theory this setup can be used for ultra-fast file sharing etc. My primary interest was getting my hands on Infiniband and OFED stack (and ultimately using it from Java). Unfortunately Socket Direct Protocol (SDP) available in Java since version 7 is a) deprecated in the latest version of OFED and b) seems to be unsupported by Java on Windows anyway. There are various libraries that provide RDMA to Java using JNI wrappers.

Monday, December 5, 2011

My CPU collection

Monday, October 11, 2010

ShouldNotReachHere

#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (classFileParser.cpp:3161), pid=3136, tid=4676
# Error: ShouldNotReachHere()
#

Sunday, October 3, 2010

My first Android app





I created my first Android app. It converts a date in Gregorian Calendar into Maya "Long Count" calendar. Contrary to popular view Mayan calendar doesn't end on December 21st, 2012. You can test this in my app :-).

Sources can be found here.

Sunday, June 6, 2010

Alternative to Thread.sleep()

Follow-up to my previous post. Here is an alternative to Thread.sleep() that uses spin-yield:

private static final long SLEEP_PRECISION = TimeUnit.MILLISECONDS.toNanos(2); //TODO: Determine for current machine

public static void sleepNanos (long nanoDuration) throws InterruptedException {
final long end = System.nanoTime() + nanoDuration;


long timeLeft = nanoDuration;
do {
if (timeLeft > SLEEP_PRECISION)
Thread.sleep (
1);
else

Thread.sleep (
0); // Thread.yield();
timeLeft = end - System.nanoTime();

if (Thread.interrupted())
throw new InterruptedException ();

}
while (timeLeft > 0);

}



Test


I run 4 threads requesting 5 millisecond sleep 1000 times each (on my Dual-Core CPU). The first chart shows sleepNanos(TimeUnit.MILLISECONDS(5).toNanos()):

Actual sleep time of sleepNanos(5000000)

The second chart shows Thread.sleep(5):

Actual sleep time of Thread.sleep(5)

As you can see, sleepNanos() is much more precise. I found that this approach was originally used by Ryan Geiss for WinAmp visualization plugin.

UPDATE



Even better precision can be achieved if you are willing to consume more CPU power doing Spin-Wait for the last part of the wait:







public static void sleepNanos (long nanoDuration) throws InterruptedException {
final long end = System.nanoTime() + nanoDuration;
long timeLeft = nanoDuration;
do {

if (timeLeft > SLEEP_PRECISION)
Thread.sleep (1);
else
if (timeLeft > SPIN_YIELD_PRECISION)

Thread.sleep(0);

timeLeft = end - System.nanoTime();

if (Thread.interrupted())
throw new InterruptedException ();

} while (timeLeft > 0);

}

Thursday, May 13, 2010

Measuring nanoseconds in Java / Windows

I was under false impression that interval timers in Java allow sub-millisecond precision on Windows. I knew that Thread.sleep(millis, nanos) internally uses milliseconds, but for some reason I thought that methods like LockSupport.parkNanos() method provide precise waits. Well, I was wrong. The smallest delay this method can realize is approximately 1.95 milliseconds on my Windows PC.



Back in 2006 David Holmes explained that Java timer intervals are based on waitForMultipleObjects() Windows API (which uses dwMilliseconds). They still are.



One simple (but not universal) workaround is a "spin-sleep":


private static void sleepNanos (long nanoDelay) {
final long end = System.nanoTime() + nanoDelay;
do {
Thread.yield(); // Thread.sleep (0);
} while (System.nanoTime() < end);
}


When running standalone this method consumes all free resources of single CPU core, but it will share with other threads that may be running. For large durations it can be enhanced to use Thread.sleep() for bulk of the waiting.



P.S. Difference between Thread.yield() and Thread.sleep( 0 ) is explained here.

Friday, December 11, 2009

Old trap

I was puzzled why one of my objects had incorrect toString()

class Foo { 
private long id;
private String name;

Foo(
long id, String name) {
this.id = id;
this.name = name;
}

public String toString () {
return id + ' ' + name;
}
}


The statement new Foo(123, "Hello") prints something like "155Hello" instead of "123 Hello". Of course the reason is pretty simple:

 
public String toString2() {
return new StringBuilder().append(id + 32L).append(name.toString()).toString();
}